System and method for analyzing corneal lesion using anterior ocular segment image, and computer-readable recording medium

ABSTRACT

A system for and a method of analyzing a corneal lesion using an anterior segment image according to the present invention. The system includes: an image acquisition unit configured to acquire an anterior segment image from the eyeball of a subject, a feature extractor configured to extract feature information on a position and a cause of a lesion in the cornea from the anterior segment image by applying a convolution layer to the anterior segment image through machine learning on the basis of a database in which clinical information pre-acquired by analyzing positions and causes of lesions in the corneas of subjects is stored; and a result determination unit configured to identify a position of the cornea from the anterior segment image using the feature information and to analyze and determine the position and the cause of the lesion in the cornea from the position of the cornea.

TECHNICAL FIELD

The present invention relates to a system for and a method of analyzinga corneal lesion using an anterior segment image and, particularly, to asystem for and a method of analyzing a position and a cause of an oculardisease in an anterior segment image by performing machine learning ofclinical information of a subject on the basis of deep learning.

BACKGROUND ART

The cornea is the outermost surface of the eyeball and is the colorlesstransparent outer covering at the front of the eyeball. The cornea iscomposed of five layers: epithelium, Bowman's membrane, stroma,Descemet's membrane, and endothelium in order from the outermost surfaceinward. Stroma is composed of flat fibroblasts and a collagen fiberhaving an arrangement of regular layers in the shape of a plate.

The cornea is the part of the eye through which light first passes, andblood vessels are not distributed in the cornea. In addition, thecorneal surface is kept wetted with tears. The tears serve to constantlytransmit and refract light, and thus an ordinary person can maintain20/20 vision or better.

The cornea covers the front portion of the eye, and thus may be subjectto damage due to trauma, infection, and other immune reactions. As aresult, the cornea may lose properties of the transparent cornea,leading to corneal opacity, or a new blood vessel may be formed, leadingto serious vision loss. Keratitis is a disease that causes inflammationof the cornea that results from an infection or a non-infection. Whennot treated properly, keratitis causes damage to corneal tissue andincreases the risk of decreased vision due to the occurrence of cornealopacity.

Particularly, because developing keratitis may cause permanent visionloss, an earlier diagnosis and treatment thereof are very important.Keratitis causes hyperemia, a feeling of being in contact with a foreignsubstance, and a pain. Symptoms of keratitis are sensitiveness to light,excessive tears, and blurred vision. Keratitis may be associated with aninfection or a non-infection. Examples of non-infectious keratitisinclude aseptic keratitis due to a corneal immune reaction, toxickeratitis due to a drug, neurotrophic keratitis due to damage to acorneal nerve, and the like. Infectious keratitis is caused by infectionof the cornea with various of pathogens, such as bacteria, viruses, andfungi. Bacteria causing infectious keratitis include staphylococcus orstreptococcus that is gram-positive and pseudomonsa aeruginosa that isgram-negative. Viruses causing infectious keratitis include herpessimplex virus. Fungi causing infectious keratitis include fusarium andcandida.

More than 2.3 million people per year in Korea receive a hospitaltreatment for keratitis, and the total amount of medical expensesincurred as of 2018 amounted to more than KRW 110 billion. In addition,with an increase in the use of contact lenses, an incidence rate ofkeratitis is expected to continuously increase. However, a diagnosiswith keratitis depends entirely on the knowledge and experience ofophthalmologists.

An ophthalmologist makes a diagnosis by directly observing an anteriorsegment of eye using a slit lamp microscope or cultivates a cornealspecimen for identifying a cause of keratitis.

The golden standard is the cultivation of the corneal specimen. However,it takes a long time to cultivate the corneal specimen, with the resultthat a cultivation rate is low. In an actual clinical practice, atreatment for keratitis cannot be delayed until a result of the specimencultivation is available. Therefore, the ophthalmologist first providesan empirical treatment based on the result of identifying a shape, size,interface, position, and the like of a lesion using the slit lampmicroscope. Then, a treatment based on the result of the cultivationspecimen is provided instead of the previous treatment method.

However, infectious keratitis caused by bacteria, fungi, amoebas, andviruses and non-infectious keratitis are not clearly distinguished fromeach other in a clinical manner. Due to an erroneous diagnosis, anunnecessary treatment may be provided for a long time or a requiredtreatment may not be provided at an appropriate time. In this case,without suffering from decreased vision, a patient may have vision thatwould have been improved if an accurate diagnosis and a quick treatmentwere provided at an appropriate time. Moreover, there occurs a problemin that the patient's quality of life may be decreased and that theamount of medical expenses may be increased.

SUMMARY OF THE INVENTION Technical Problem

An object of the present invention, which is made to solve the problemas mentioned above, is to provide a system for analyzing a corneallesion using an anterior segment image in such a manner that an accuratequick diagnosis is provided through machine learning.

Another object of the present invention is to provide a method ofanalyzing a corneal lesion using an anterior segment image in order toincrease the accuracy of a diagnosis in a process of applying machinelearning.

Still another object of the present invention is to provide acomputer-readable storage device on which to store a program thatperforms a method of analyzing a corneal lesion that is realized usingthe above-mentioned system.

Technical Solution

In order to accomplish the above-mentioned objects, according to anaspect of the present invention, there is provided a system foranalyzing a corneal lesion using an anterior segment image, the systemincluding: an image acquisition unit configured to acquire an anteriorsegment image from the eyeball of a subject; a feature extractorconfigured to extract feature information on a position and a cause of alesion in the cornea from the anterior segment image by applying aconvolution layer to the anterior segment image through machine learningon the basis of a database in which clinical information pre-acquired byanalyzing positions and causes of lesions in the corneas of subjects isstored; and a result determination unit configured to identify aposition of the cornea from the anterior segment image using the featureinformation and to analyze and determine the position and the cause ofthe lesion in the cornea from the position of the cornea.

In the system, the feature extractor may include a residual network(ResNet) obtained by stacking at least one network of a plurality ofnetworks each including a convolution layer, a pooling layer, and anactivation function or a rectified linear unit (ReLU) function and theResNet may extract the feature information, including a multiple-channelfeature map for extracting a suspicious region, from the anteriorsegment image.

In the system, the feature extractor may include a lesion guiding moduleconfigured to extract a lesion region more precise than the suspiciousregion through convolution of the feature information and positionaldata of a lesion in the clinical information, the positional datarepresenting a position of the lesion.

In the system, the feature extractor may include a slit lamp maskadjustment module configured to adjust a slit beam image present in aregion of the anterior segment image.

In the system, the slit lamp mask adjustment module may exclude a slitlamp portion from the anterior segment image and may cause the positionof the cornea or the position of the lesion in the cornea, which isincluded in the anterior segment image, to be learned.

In the system, the slit lamp mask adjustment module may apply theconvolution layer to a 3^(rd) label feature vector of the anteriorsegment image and then may adjust a masking ratio by applying aweighting factor for a slit beam portion.

In the system, the result determination unit may re-input a 3^(rd) labelfeature vector output from the feature extractor and a prediction vectorcorresponding to a 2^(nd) label feature vector computed by applying afully connected layer to the 3^(rd) label feature vector, into the fullyconnected layer and thus may identify the cause of the lesion in thecornea.

According to a second aspect of the present invention, there is provideda method of analyzing a corneal lesion using an anterior segment image,the method of processing steps realized by constituent elements of asystem for analyzing a corneal lesion, the method including: a step ofacquiring, by an image acquisition unit, an anterior segment image fromthe eyeball of a subject; a step of extracting, by a feature extractor,feature information on a position and a cause of a lesion in the corneafrom the anterior segment image by applying a convolution layer to theanterior segment image and a database in which clinical informationpre-acquired by analyzing positions and causes of lesions in the corneasof subjects is stored; and a step of analyzing and determining, by aresult determination unit, the position and the cause of the lesion inthe cornea after identifying a position of the cornea from the anteriorsegment image using the feature information.

In the method, the step of extracting, by the feature extractor, thefeature information may include a step of extracting the featureinformation including a multiple-channel feature map for extracting asuspicious region, from the anterior segment image, the featureinformation being destined for a residual network (ResNet) obtained bystacking at least one network of a plurality of networks each includinga convolution layer, a pooling layer, and an activation function or arectified linear unit (ReLU) function.

In the method, the step of extracting the feature information mayinclude a step of extracting a lesion region more precise than thesuspicious region through convolution of the feature information andpositional data of a lesion in the clinical information, the positionaldata representing a position of the lesion, by a lesion guiding module(LGM).

In the method, the step of extracting, by the feature extractor, thefeature information may include a step of adjusting, by a slit lamp maskadjustment module (MAM) of the feature extractor, a slit beam imagepresent in a region of the anterior segment image.

In the method, the step of adjusting, by the slit lamp mask adjustmentmodule, the slit beam image may include a step of applying, the slitlamp mask adjustment module, the convolution layer to the anteriorsegment image and an already-collected slit lamp region mask; and a stepof adjusting a masking ratio by applying a weighting factor for a slitbeam portion.

In the method, the step of analyzing and determining, by the resultdetermination unit, the position and the cause of the lesion in thecornea may include: a step of outputting, by the feature extractor, a3rd label feature vector; a step of outputting a prediction vectorcorresponding to a 2^(nd) label feature vector computed by applying afully connected layer to the 3rd label feature vector; and a step ofre-inputting, by the result determination unit, the feature vector andthe prediction vector to the fully connected layer.

According to still another aspect of the present invention, there isprovided a computer-readable storage device on which a program thatperforms the method of analyzing a corneal lesion is recorded.

Advantageous Effects

The system for and the method of analyzing a corneal lesion according tothe present invention, which are confingured as described above, canprovide the advantageous effect of lowering a misdiagnosis rate throughan accurate quick diagnosis using a diagnosis model that results fromlearning on the basis of clinical information.

The system and the method according to the present invention can providethe advantageous effect of making a suitable diagnosis consistent with acurrent trend on the basis of a database continually updated becausesusceptibility of a causative organism to an antibiotic tends to changewith time.

The system and the method according to the present invention can beutilized when an ophthalmologist makes a diagnosis as keratitis, andthus can provide the advantageous effect of possibly improving access tomedical services in an area where a few licensed medical facilities arelocated and where the ophthalmologist does not do his/her medicalpractice or access of those living in poverty to the medical services.

BRIEF DESCRIPTION OF DRAWINGS

FIG. 1 is a block diagram illustrating a system for analyzing a corneallesion using an anterior segment image according to the presentinvention.

FIG. 2 is a schematic diagram illustrating that a result of a diagnosisis derived in the system for analyzing a corneal lesion using ananterior segment image according to the present invention.

FIG. 3 is a diagram schematically illustrating the system for analyzinga corneal lesion using an anterior segment image according to thepresent invention.

FIG. 4 is a diagram illustrating a learning process in the system foranalyzing a corneal lesion according to an embodiment of the presentinvention.

FIG. 5 is a flowchart illustrating the method of analyzing a corneallesion using an anterior segment image according to the presentinvention.

FIG. 6 is a block diagram illustrating a feature extractor and a lesionguiding module according to an embodiment of the present invention andprocessing that is performed by the feature extractor and the lesionguiding module.

FIG. 7 is a block diagram illustrating the feature extractor and a slitlamp mask adjustment module according to an embodiment of the presentinvention and processing that is performed by the feature extractor andthe slit lamp mask adjustment module.

FIG. 8 is a diagram illustrating a process in which according to anembodiment of the present invention, the slit lamp mask adjustmentmodule adjusts an original mask by applying a weighting factor to anoriginal mask.

FIG. 9 is photographs for clinical information reference on bacterialkeratitis among anterior segment images according to an embodiment ofthe present invention.

FIG. 10 is photographs showing the clinical information resulting from amedical specialist analyzing positions and causes of lesions in theanterior segment image in FIG. 8 in order to build a database in whichthe clinical information is stored.

FIG. 11 is photographs showing a fungal keratitis in the anteriorsegment image according to an embodiment of the present invention.

FIG. 12 is photographs showing the clinical information resulting fromthe medical specialist analyzing the positions and the causes of thelesions in the anterior segment image in FIG. 10 to build the databasein which the clinical information is stored.

FIG. 13 is a diagram illustrating that contents obtained by analyzingthe clinical information in the anterior segment image in FIGS. 9 to 12are produced as a keyword set.

FIG. 14 is a view illustrating a screen on which the keyword set in FIG.13 is displayed for outputting when a program runs according to anembodiment of the present invention.

DETAILED DESCRIPTION OF THE INVENTION Best Mode

In order to accomplish the above-mentioned objects, according to anaspect of the present invention, there is provided a system foranalyzing a corneal lesion using an anterior segment image, the systemincluding: an image acquisition unit configured to acquire an anteriorsegment image from the eyeball of a subject; a feature extractorconfigured to extract feature information on a position and a cause of alesion in the cornea from the anterior segment image by applying aconvolution layer to the anterior segment image through machine learningon the basis of a database in which clinical information pre-acquired byanalyzing positions and causes of lesions in the corneas of subjects isstored; and a result determination unit configured to identify aposition of the cornea from the anterior segment image using the featureinformation and to analyze and determine the position and the cause ofthe lesion in the cornea from the position of the cornea.

In the system, the feature extractor may include a residual network(ResNet) obtained by stacking at least one network of a plurality ofnetworks each including a convolution layer, a pooling layer, and anactivation function or a rectified linear unit (ReLU) function and theResNet may extract the feature information, including a multiple-channelfeature map for extracting a suspicious region, from the anteriorsegment image.

In the system, the feature extractor may include a lesion guiding moduleconfigured to extract a lesion region more precise than the suspiciousregion through convolution of the feature information and positionaldata of a lesion in the clinical information, the positional datarepresenting a position of the lesion.

In the system, the feature extractor may include a slit lamp maskadjustment module configured adjust a slit beam image present in aregion of the anterior segment image.

In the system, the slit lamp mask adjustment module may exclude a slitlamp portion from the anterior segment image and may cause the positionof the cornea or the position of the lesion in the cornea, which isincluded in the anterior segment image, to be learned.

In the system, the slit lamp mask adjustment module may apply theconvolution layer to a 3^(rd) label feature vector of the anteriorsegment image and then may adjust a masking ratio by applying aweighting factor for a slit beam portion.

In the system, the result determination unit may re-input a 3^(rd) labelfeature vector output from the feature extractor and a prediction vectorcorresponding to a 2^(nd) label feature vector computed by applying afully connected layer to the 3^(rd) label feature vector, into the fullyconnected layer and thus may identify the cause of the lesion in thecornea.

According to another aspect of the present invention, there is provideda method of analyzing a corneal lesion using an anterior segment image,the method of processing steps realized by constituent elements of asystem for analyzing a corneal lesion, the method including: a step ofacquiring, by an image acquisition unit, an anterior segment image fromthe eyeball of a subject; a step of extracting, by a feature extractor,feature information on a position and a cause of a lesion in the corneafrom the anterior segment image by applying a convolution layer to theanterior segment image and a database in which clinical informationpre-acquired by analyzing positions and causes of lesions in the corneasof subjects is stored; and a step of analyzing and determining, by aresult determination unit, the position and the cause of the lesion inthe cornea after identifying a position of the cornea from the anteriorsegment image using the feature information.

In the method, in the step of extracting, by the feature extractor, thefeature information may include a step of extracting the featureinformation including a multiple-channel feature map for extracting asuspicious region, from the anterior segment image, the featureinformation being destined for a residual network (ResNet) obtained bystacking at least one network of a plurality of networks each includinga convolution layer, a pooling layer, and an activation function or arectified linear unit (ReLU) function.

In the method, the step of extracting the feature information mayinclude a step of extracting a lesion region more precise than thesuspicious region through convolution of the feature information andpositional data of a lesion in the clinical information, the positionaldata representing a position of the lesion, by a lesion attentionmodule.

In the method, the step of extracting, by the feature extractor, thefeature information may include a step of adjusting, by a slit lamp maskadjustment module of the feature extractor, a slit beam image present ina region of the anterior segment image.

In the method, the step of adjusting, by the slit lamp mask adjustmentmodule, the slit beam image may include a step of applying, the slitlamp mask adjustment module, the convolution layer to the anteriorsegment image and an already-collected slit lamp region mask; and a stepof adjusting a masking ratio by applying a weighting factor for a slitbeam portion.

In the method, the step of analyzing and determining, by the resultdetermination unit, the position and the cause of the lesion in thecornea may include: a step of outputting, by the feature extractor, a3^(rd) label feature vector; a step of outputting a prediction vectorcorresponding to a 2^(nd) label feature vector computed by applying afully connected layer to the 3^(rd) label feature vector; and a step ofre-inputting, by the result determination unit, the feature vector andthe prediction vector to the fully connected layer.

According to still another aspect of the present invention, there isprovided a computer-readable storage device on which a program thatperforms the method of analyzing a corneal lesion is recorded.

Mode for Invention

The terms to be used throughout the present specification are brieflydescribed, and the present invention is in detail described.

The terms to be used throughout the present specification are selectedfrom among general terms that are currently used as widely as possible,considering a functional meaning in the present specification. However,the terms may vary depending on the intention of a person of ordinaryskill in the art, a judicial precedent, the appearance of a newtechnology, or the like. In addition, as a special case, there is a termthat is arbitrarily selected by the applicant. When such a term is used,the meaning thereof will be in detail interpreted. Therefore, themeaning of the term used throughout the present specification should bedefined in light of the present specification, instead of beinginterpreted as indicated by the name of the term.

Unless otherwise described, the expression “includes a constituentelement,” when used throughout the present specification, means ″ mayfurther include any other constituent element, not “excluding any otherconstituent element.” In addition, the term “unit,” “module,” or thelike, which is used throughout the specification, means an individualphysical entity that performs at least one function or operation and maybe realized in hardware, software, or a combination of both. Inaddition, a constituent element, when referred to as being “connectedto” one other constituent element, may be “directly connected to” oneother constituent element or may be “indirectly connected to” one otherconstituent with an intervening constituent element in between.

An embodiment of the present invention will be described below insufficient detail to enable a person of ordinary skill in the art towhich the present invention pertains to practice the present inventionwithout undue experimentation. However, the present invention may bepracticed in various different forms and is not limited to theembodiment described below. For definite description of the embodimentof the present invention, a portion not associated therewith is omittedfrom the drawings, and the same constituent elements are given the samereference numeral throughout the present specification.

FIG. 1 is a block diagram illustrating a system 1 for analyzing acorneal lesion using an anterior segment image according to the presentinvention.

With reference to FIG. 1 , the system 1 according to the presentinvention includes an image acquisition unit 10, a feature extractor 20,and a result determination unit 30. The feature extractor 20 includes alesion guiding module 21 that operates in conjunction with a slit lampmask adjustment module 31 and a database, receives positional data 211of a lesion, as an input, and learns a position of the lesion.

The image acquisition unit 10 may acquire the anterior segment imagefrom an eyeball of the subject. The anterior segment image may beacquired through a slit lamp microscope in a doctor's office, ahospital, and the like or through the image acquisition unit 10 of auser's device or like. The user devices may include any devices, such asa smartphone, a tablet PC, and a digital camera, that include an imagingmodule and is capable of performing imaging.

The feature extractor 20 may extract feature information on the positionand a cause of the lesion in the cornea from the anterior segment imageby applying a convolution layer to the anterior segment image throughmachine learning on the basis of the database in which clinicalinformation pre-acquired by analyzing positions and causes of lesions inthe corneas of subjects is stored.

The result determination unit 30 may identify a position of the corneafrom the anterior segment image using the feature information and mayanalyze and determine the position and the cause of the lesion in thecornea from the position of the cornea.

A machine learning model according to an embodiment of the presentinvention may include a deep neural network (DNN). The deep neuralnetwork refers to a neural network having a deep structure. The deepneural network is caused to learn a large amount of data to be learnedin a structure made up of a multi-layered network and thus toautomatically learn a feature of a signal and a relationship betweensignals. Accordingly, learning is performed on a network for diagnosingthe anterior segment image.

Particularly, in order to determine a result of the diagnosis, aconvolution neural network (CNN) may be used as a model for extractingthe feature information from the anterior segment image.

FIG. 2 is a schematic diagram illustrating that the result of thediagnosis is derived in the system 1 for analyzing a corneal lesionusing an anterior segment image according to the present invention.

With reference to FIG. 2 , the result determination unit 30 may re-inputa 3^(rd) label feature vector output from the feature extractor 20 and aprediction vector corresponding to a 2^(nd) label feature vectorcomputed by applying a fully connected layer to the 3^(rd) label featurevector, into the fully connected layer and thus may identify the causeof the lesion in the cornea.

FIG. 2 is a diagram illustrating the entire system 1 for analyzing acorneal lesion, the system 1 utilizing the anterior segment image and a2^(nd) label (a keyword set). When the anterior segment image is inputinto the feature extractor 20, 3^(rd) label vector feature information(image feature) is extracted. This feature information becomes a 2^(nd)label vector after passing through the fully connected layer.

The feature extractor 20 is configured to have the shape that resultsfrom stacking multiple convolutional layers on top of each other.Because of this, it is possible that an intermediate image feature isobtained between layers. The intermediate image feature is caused topass through another deep learning network predicting a 2^(nd) label,and thus a 2^(nd) label prediction vector may be obtained.

A vector obtained from the above-described image feature and the 2^(nd)label prediction vector are added up, and the sum thereof is also causedto pass through the fully connected layer. Thus, a cause (bacterial orfungal) of keratitis that is a final result of the diagnosis may beobtained.

During a learning process in the entire system 1, a set of [an anteriorsegment image, result-of-diagnosis data, and 2^(nd) label data] is used,and when the set is actually used, only the anterior segment image isinput.

FIG. 3 is a diagram schematically illustrating the system 1 foranalyzing a corneal lesion using an anterior segment image according tothe present invention.

With reference to FIG. 3 , the feature extractor 20 may include aresidual network (ResNet) obtained by stacking at least one network of aplurality of networks each including a convolution layer, a poolinglayer, and an activation function or a rectified linear unit (ReLU)function, and the ResNet may extract the feature information, includinga multiple-channel feature map for extracting a suspicious region, fromthe anterior segment image.

The lesion guiding module 21 is inserted between each of the ResNetlayers. According to an embodiment of the present invention, the lesionguiding module 21 may go through a process four times. Regarding thefeature extractor 20, a process for going through the ResNet layers fourtimes in total may be expressed as the feature extractor 20. Thisprocess of processing data will be described below, together with amethod of analyzing a corneal lesion.

According to an embodiment of the present invention, when the anteriorsegment image is input while in use, a lesion indication image may beobtained in the lesion guiding module (LG module) 21 inside the featureextractor 20. The 2^(nd) label vector and the 2^(nd) label predictionvector that pass through the feature extractor 20 and the fullyconnected layer may be concatenated.

Consequently, the result determination unit 30 may analyze and determinethe position and the cause of the lesion in the cornea. This process mayinclude a step of outputting, by the feature extractor 20, a 3^(rd)label feature vector, a step of outputting a prediction vectorcorresponding to a 2^(nd) label feature vector computed by applying afully connected layer to the 3^(rd) label feature vector, and a step ofre-inputting, by the result determination unit 30, the feature vectorand the prediction vector into the fully connected layer.

The convolution neural network (CNN) is configured as illustrated n FIG.3 . According to an embodiment of the present invention, the featureextractor 20 may extract a feature of the anterior segment image, and,on the basis of the extracted feature, the result determination unit 30may analyze and determine what causes a disease in the anterior segmentimage.

The feature extractor 20 may be configured with the convolution layerand the pooling layer (not illustrated), and a disease analysis unit 300may be designed as the fully connected layer.

Specifically, in the convolution layer of the feature extractor 20, afeature map (not illustrated) may be formed by applying a plurality offilters in each region of the anterior segment image, and in the poolinglayer (not illustrated), the feature map may be resized. In addition,the feature extractor 20 may be formed in such a manner that severalconvolution layer and several pooling layer are alternately arranged.

According to another embodiment of the present invention, instead of thefully connected layer (not illustrated), a classification model, such asa multiple-layer perception (MLP) or a support vector machine (SVM), maybe included in the result determination unit 30. Thus, a prediction maybe made through classification of the extracted features.

FIG. 4 is a diagram illustrating the learning process in the system 1for analyzing a corneal lesion according to an embodiment of the presentinvention. FIG. 4 is a diagram that results from combining the diagramsin FIGS. 2 to 4 to illustrate in detail the learning process in thesystem 1 for analyzing a corneal lesion according to the presentinvention.

With reference to FIG. 4 , the anterior segment image and a result of adiagnosis by the slit beam mask in which a region of the anteriorsegment image on which the slit lamp sheds light is marked may bepresent as data used in the learning process. The data used in thelearning process may be expressed as a set of [an anterior segmentphotograph, a slit beam mask, result-of-diagnosis correct answer (acause of keratitis), and a 2^(nd) label correct answer]. At this point,only the anterior segment image may be used as an input in the system 1,and the other data are used to compute a loss function necessary in theleaning process.

The feature extractor 20 may include the lesion guiding module 21 andthe slit lamp mask adjustment module 31. The lesion guiding module 21,as described above, may perform convolution of the positional data 211of the lesion, which contains the position of the lesion in the cornea,and the 3^(rd) label feature vector and thus may compute the position ofthe lesion. The slit lamp mask adjustment module 31 may learn theanterior segment image, as clinical information, by adjusting a slitbeam region present in a region of the anterior segment image.

Deep convolutional generative adversarial nets (DCGAN) may be used inorder to accurately determine positional information of the lesion.However, according to an embodiment of the present invention, supervisedlearning based on the positional data 211 of the lesion that are markedby a medical specialist may be performed on the feature extractor 20.

The feature extractor 20 may include the ResNet and the lesion guidingmodule 21. During the learning process, the lesion guiding module 21 mayserve to emphasize a feature map of a corresponding region in such amanner that the feature extractor 20 pays attention to the lesion.

At this point, the positional data 211 of the lesion (spatial attentionground truth (GT)) that are marked by the medical specialist areprovided in such a manner as to serve as a correct answer for thesupervised learning. The learning is performed in the direction ofmaking a lesion segment extracted by the network consistent with theprovided positional data 211 of the lesion.

The slit lamp mask adjustment module 31 may be caused to learn theposition of the cornea or the position of the lesion in the cornea,which is included in the anterior segment image, with a slit lampportion being excluded from the anterior segment image.

In addition, the slit lamp mask adjustment module 31 may adjust amasking region ratio by applying the convolution layer to the anteriorsegment image and the slit beam mask in which the region of the anteriorsegment image on which the slit lamp sheds light is marked. This maskingprocessing process will be described below, together with the method ofanalyzing a corneal lesion.

The slit lamp mask adjustment module 31 is used only during the learningprocess in the system 1. An object of the slit lamp mask adjustmentmodule 31 is to extract feature points of two different types of imageinput, that is, feature points of a broad beam and a slit beam.

The number of weighting factors used to perform the learning on thefeature extractor 20 is limited, and because of this, a capacity forlearning various feature points is also limited. Therefore, the morevarious types of images and the more various types feature points thereare, the more difficult problem occurs. Thus, the accuracy can bereduced.

In the experimental example of the present invention, a case where abroad beam image and a slit beam image were learned separately from eachother and a case where the broad beam image and the slit beam image werelearned together were compared with each other. The result showed that,although more learning images were provided in the latter case than inthe former case, there is seldom sufficient difference in accuracybetween the former and latter cases.

In order to solve this problem, the slit lamp mask adjustment module 31(used as a synonym for the slit lamp mask adjustment module) may preventa slit beam portion that is expected to be extracted, as a featurepoint, by the system 1, but does not include the lesion, from beinglearned. Thus, the learning can be efficiently performed.

According to an embodiment of the present invention, color informationof the lesion may include discoloration information of the cornea, theconjunctiva, or the sclera. For example, in a case where the cornea isinfected, a color of the detected lesion can be expressed as white.

In addition, in the case of a staining photograph showing the infectedcornea, anterior segment diseases may be classified through colorinformation of the lesion, such as when a color of the detected lesionportion is expressed as green. Moreover, the anterior segment disease isdetermined, depending on a state of a blood vessel around the cornea.Therefore, blood-vessel information of a neighboring segment of thecornea may be feature information. Furthermore, like surface informationof the conjunctiva or the sclera, the anterior segment disease can beclassified according to surface roughness, surface smoothness, or thelike. Moreover, the anterior segment diseases of eye may also beclassified by each of the above-described pieces of feature information.However, among the above-described pieces of feature information, theanterior segment diseases of eye can be classified considering at leasttwo pieces of feature information.

According to another embodiment of the present invention, the deepconvolutional generative adversarial nets (DCGAN) may be included in themachine learning model of the feature extractor 20. The resultdetermination unit 30 may output the position and the cause of thelesion, together with image information of the lesion that is extractedon the basis of the anterior segment image and the machine learningmodel.

FIG. 5 is a flowchart illustrating the method of analyzing a corneallesion using an anterior segment image according to the presentinvention.

With reference to FIG. 5 , the method of analyzing a corneal lesionaccording to the present invention may include Sep S10 of acquiring ananterior segment image, Step S20 of extracting feature information, andStep S30 of determining a position and a cause of a lesion.

In Step S10 of acquiring an anterior segment image, the anterior segmentimage is acquired by the image acquisition unit 10 from the eyeball ofthe subject.

In Step S20 of extracting feature information, the feature extractor 20extracts the feature information on the position and the cause of thelesion in the cornea from the anterior segment image by applying theconvolution lay to the anterior segment image and the database in whichthe clinical information pre-acquired by analyzing the positions andcauses of the lesions in the corneas of the subjects is stored.

In Step S30 of determining a position and a cause of a lesion, theresult determination unit 30 identifies the position of the cornea inthe anterior segment image using the feature information and analyzesand determines the position and the cause of the legion in the cornea.

FIG. 6 is a block diagram illustrating the feature extractor 20 and thelesion guiding module 21 according to an embodiment of the presentinvention and processing that is performed by the feature extractor 20and the lesion guiding module 21.

With reference to FIG. 6 , Step S20 of extracting feature informationmay include a step of extracting, the feature extractor 20, the featureinformation including a multiple-channel feature map in which a lesionregion important for a diagnosis is emphasized, from the anteriorsegment image through the residual network (ResNet) having the pluralityof layers and the lesion guiding module 21.

The ResNet has a network structure that is used universally in the fieldof deep learning, and the lesion guiding module 21 may be insertedbetween each of four layers of the ResNet. The lesion guiding module 21uses the 3^(rd) label vector image feature, as an input, and this imagefeature represents an output value of each layer of the ResNet.

A label of the image feature is lowered while the image feature passesthrough the convolution layer inside the lesion guiding module 21 andthus, the image feature becomes a 2^(nd) label vector spatial attention.The spatial attention may function as a type of filter and may serve toincrease a feature value of the important lesion segment by performingan element-wise multiplication arithmetic operation on the imagefeature.

The image feature in which the lesion segment is emphasized also goesthrough an operation of performing element-wide addition to an originalimage feature, and becomes an output of the lesion guiding module 21.

Step of extracting the feature information that is destined for thelesion guiding module 21 may include Step of performing convolution ofthe positional data 211 of the lesion that contains the position of thelesion in the cornea, and the 3^(rd) label feature vector.

According to an embodiment of the present invention, an output of theResNet layer and the lesion guiding module 21 may also be used as aninput of the ResNet layer. The feature that passes through all fourlayers of the ResNet that serve as the feature extractor 20 may passthrough the fully connected layer in which the result determination unit30 finally performs disease analysis. Then, a result of a finaldiagnosis as keratitis may be output.

During the learning process, the positional data 211 of the lesion(spatial attention GT) that are marked by the medical specialist may beinput into the lesion guiding module 21. The learning may be performedon the lesion guiding module 21 in the direction of ensuring consistencywith the positional data 211 of the lesion (spatial attention is spatialattention GT) and accurately predicting the result of the diagnosis.

FIG. 7 is a block diagram illustrating the feature extractor 20 and theslit lamp mask adjustment module 31 according to an embodiment of thepresent invention and processing that is performed by the featureextractor 20 and the slit lamp mask adjustment module 31.

With reference to FIG. 7 , the method may proceed to Step of adjusting,by the slit lamp mask adjustment module 31 of the feature extractor 20,the slit beam image present in the region of the anterior segment image

The slit lamp mask adjustment module 31 (mask adjusting module) is amodule that is used only for the learning process in the system 1 foranalyzing a corneal lesion. The slit lamp mask adjustment module 31operates in such a manner that leaning image data in which the broadbeam image and the slit beam image are both present are efficientlylearned.

An input of the slit lamp mask adjustment module 31 is a mask photographshowing a position of the slit lamp in the anterior segment image. Anoutput thereof is a photograph in the form of a mask, but variesslightly from a first input mask photograph in terms of contents.

The learning process may be performed in the direction of reducing threeloss functions.

1) Cross entropy loss is a loss function that is used universally indeep learning. The more accurate the result of the diagnosis, the moredecreased a value thereof.

2) Mask difference loss is a loss function that sets a mask passingthrough the slit lamp mask adjustment module 31 and an input mask insuch a manner as not to vary widely from each other. The more similarthe input mask and an output mask are to each other, the more decreaseda value thereof.

3) Softmax difference loss is a loss function that represents adifference between the results of the diagnosis. During the learningprocess, one slit beam image passes two times through the system 1 foranalyzing a corneal lesion. At this point, the first time the slit beamimage passes, an original photograph may be used, and the second timethe slit beam image passes, the anterior segment image whose portionmarked by the mask passing through the slit lamp mask adjustment module31 is excluded may be used.

The image feature passing through the ResNet changes into a vector witha size of 2×1 while passing through the fully connected layer in whichthe result determination unit 30 performs the disease analysis.

Values of the vectors are referred to as softmax scores, and may becomputed as a score indicating whether the input anterior segment imagerepresents bacterial or fungal keratitis.

As a softmax score output from a photograph in which the slit lampportion is hidden and a softmax score output from the originalphotograph become more similar to each other, the slit lamp portionresults in having less influence on the result of the diagnosis. Usingthe softmax difference loss principle, the learning may be performed onthe network in the direction of not considering the slit lamp portion asbeing important.

As the softmax score output from the photograph in which the slit lampportion is hidden and the softmax score output from the originalphotograph becomes more similar to each other, the value of the lossfunction is more decreased.

FIG. 8 is a diagram illustrating a process in which according to anembodiment of the present invention, the slit lamp mask adjustmentmodule 31 adjusts an original mask 311 by applying a weighting factor tothe original mask 311.

With reference to FIG. 8 illustrating the inside of the slit lamp maskadjustment module 31, the process may include Step of applying, by theslit lamp mask adjustment module 31, a convolution layer to an anteriorsegment image and a pre-collected slit lamp region mask and Step ofadjusting a masking region ratio by applying the convolution layer tothe anterior segment image and a slit beam mask in which a region of theanterior segment image on which the slit lamp sheds light is marked.

The anterior segment image and a mask representing a slit beam portionof a corresponding image are concatenated, and a result of theconcatenating is an input of the slit lamp mask adjustment module 31,and a 3^(rd) label vector being input passes through the convolutionlayer and becomes an M_pos mask 312 and an M_neg mask 313 that arereferred to as masks indicating how much the original mask 311 isadjusted.

That is, a portion of the original mask 311 that is represented by theM_neg mask 313 disappears, and a portion thereof that is represented bythe M_pos mask 312 appears. A finally adjusted mask may be an output ofthe slit lamp mask adjustment module 31.

The symbol ⊕ indicating that the M_pos mask 312 and the M_neg mask 313in FIG. 8 are concatenated refers to computation for adjusting theoriginal mask 311 using the M_pos mask 312 and the M_neg mask 313. Asdescribed above, the more decreased a difference between a final maskand the original mask 311, the more decreased the mask difference losshas to be. Because of this, the mask difference loss can be expressed asthe sum of weighting factors of the M_pos mask 312 and the M_neg mask313 (the mask always has a positive value).

The pre-acquired clinical information to be applied to the learningprocess according to an embodiment of the present invention will bedescribed below. The experimental example of the embodiment of thepresent invention shows the anterior segment image, the positional data211 of the lesion, and the result of the diagnosis that are reflected inthe learning process by the feature extractor 20.

FIGS. 9 to 12 are photographs showing a process of generating theclinical information according to the present invention. A process ofaccumulating the above-described database after being analyzed by adoctor or a medical specialist is shown.

FIGS. 9 and 11 are clinical-information reference photographs forbacterial keratitis (FIG. 9 ) and fungal keratitis (FIG. 11 ) in theanterior segment image according to an embodiment of the presentinvention. FIGS. 10 and 12 are photographs showing the clinicalinformation resulting from the medial specialist analyzing the positionsand the causes of the lesions in FIGS. 9 and 11 in order to build thedatabase.

That is, FIGS. 10 and 12 are photographs showing results of imagelabeling performed by three medical specialists. A process in which adistinctive lesion segment in each of the off-line generated results ofthe diagnosis is marked and in which an appearance of the legion ismarked is shown.

According to an embodiment of the present invention, anterior segmentdiseases of eye may be classified by a structure of the anterior segmentand a type of disease. Regarding the structure of the anterior segmentof eye, the anterior segment may include the cornea, the conjunctiva,the sclera, the anterior chamber, the lens, the iris, the ciliary body,and the like.

In addition, the anterior segment diseases of eye may include allinfectious inflammatory diseases associated with bacteria, viruses, andfungi, non-infectious inflammatory diseases, such as an autoimmunedisease, non-inflammatory diseases, such as degeneration, dystrophy, andkeratoconus, and diseases associated with the anterior segment of eye,such as dry eye disease, cataract, corneal erosion, deposit, edema, andcorneal opacity.

That is, types of anterior segment diseases of eye may be classifiedinto infectious and non-infectious inflammatory diseases due to thestructure of the anterior segment, and anterior segment diseases due tonon-inflammation, trauma, aging, and the like.

FIG. 13 is a diagram illustrating that contents obtained by analyzingthe clinical information in the anterior segment image in FIGS. 9 to 12are produced as a keyword set. FIG. 14 is a view illustrating a screenon which the keyword set is displayed for outputting when a programruns.

The present invention may be practiced as a computer-readable storagedevice on which a program for realizing the method of analyzing acorneal lesion is recorded and may be realized as a program. In a casewhere the anterior segment image is input, the position and the cause ofthe lesion in the cornea can be identified with the method according tothe present invention that uses machine learning.

Instructions for each lesion that are written by the medical specialistare attached to the anterior segment image, and adjectives and nous thatare frequently used in the instructions can be extracted and produced asthe keyword set. With reference to the medical specialist-writteninstructions for the lesion, labelling with the keyword corresponding toeach anterior segment image may be performed.

According to another embodiment of the present invention, medicalinformation may be included. The medical information may include atleast one of current-state information of the anterior segment of eye,future-state prediction information of the anterior segment of eye, andtreatment information for a disease class.

In addition, the current-state information of the anterior segment ofeye may be information in which a type of disease of the anteriorsegment and a disease class thereof are included together. Thefuture-state prediction information of the anterior segment of eye maybe predicted through the machine learning model. For this prediction,the machine learning model may additionally include a recurrent neuralnetwork (RNN) and a multi-layer perceptron (MLP), and the like.

The future-state prediction information may include a type of diseasethat is predicted to occur according to a current state of the anteriorsegment of eye, and the degree to which the disease develops, and mayalso include expected rejection in accordance with the current state.Treatment information for the disease class may include an anteriorsegment disease of eye resulting from the classification and thetreatment information in accordance with the disease class.

As described above, according to the present invention, a patient can bediagnosed with keratitis on the basis of the accurate diagnosis modelbuilt on the basis of a large amount of existing data, instead of beingdiagnosed depending on an ophthalmologist's personal clinical knowledgeand experience. Thus, a misdiagnosis rate can be greatly lowered, andthus prognosis of the patient's eyesight can also be improved.

Susceptibility of a causative organism of keratitis to an antibiotictends to change with time. However, the continually updated databasethat is to be applied according to the present invention may be quicklyadapted to this change, and thus a suitable diagnosis consistent with acurrent trend can be provided.

In addition, the system for and the method of analyzing a corneal lesionaccording to the present invention may be distributed to an area where afew licensed medical facilities are located and where an ophthalmologistdoes not do his/her medical practice, thereby contributing to anincrease in equal access to domestic medical service. Moreover, thesystem and method can greatly contribute to decreasing unnecessarymedical expenses incurred due to an erroneous diagnosis or treatment ofa patient with keratitis. Thus, the saved money can be used to providemedical treatments to patients suffering from other principal diseases.The distribution of the system and the method to a developing countrycan contribute to an international community development and a worldhealth improvement.

The exemplary embodiments of the present inventions are in detaildescribed above, and it would be understandable to a person of ordinaryskill in the art to which the present invention pertains that variousmodifications to the embodiments described above are possibly madewithout departing from the scope of the present invention. Therefore,the scope of the present invention should be defined not only by thefollowing claims, but also by all alterations and modifications that arederived from the concept of equivalents of the claims, without beingdefined in a manner that is limited to the embodiments described above.

1. A system for analyzing a corneal lesion using an anterior segmentimage, the system comprising: an image acquisition unit configured toacquire an anterior segment image from an eyeball of a subject; afeature extractor configured to extract feature information on aposition and a cause of a lesion in a cornea of the eyeball from theanterior segment image by applying a convolution layer to the anteriorsegment image through machine learning on the basis of a database inwhich clinical information pre-acquired by analyzing positions andcauses of lesions in the corneas of subjects is stored; and a resultdetermination unit configured to identify a position of the cornea fromthe anterior segment image using the feature information and to analyzeand determine the position and the cause of the lesion in the corneafrom the position of the cornea.
 2. The system of claim 1, wherein thefeature extractor comprises: a residual network (ResNet) obtained bystacking at least one network of a plurality of networks each includinga convolution layer, a pooling layer, and an activation function or arectified linear unit (ReLU) function and the ResNet extracts thefeature information, including a multiple-channel feature map forextracting a suspicious region, from the anterior segment image.
 3. Thesystem of claim 2, wherein the feature extractor comprises: a lesionguiding module configured to extract a lesion region more precise thanthe suspicious region through convolution of the feature information andpositional data of a lesion in the clinical information, the positionaldata representing a position of the lesion.
 4. The system of claim 1,wherein the feature extractor comprises: a slit lamp mask adjustmentmodule adjusting a slit beam image present in a region of the anteriorsegment image.
 5. The system of claim 4, wherein the slit lamp maskadjustment module excludes a slit lamp portion from the anterior segmentimage and causes the position of the cornea or the position of thelesion in the cornea, which is included in the anterior segment image,to be learned.
 6. The system of claim 4, wherein the slit lamp maskadjustment module applies the convolution layer to the anterior segmentimage and an already-collected slit lamp region mask and then adjusts amasking ratio by applying a weighting factor for a slit beam portion. 7.The system of claim 1, wherein the result determination unit re-inputs a3^(rd) label feature vector output from the feature extractor and aprediction vector corresponding to a 2^(nd) label feature vectorcomputed by applying a fully connected layer to the 3^(rd) label featurevector, into the fully connected layer and thus identifies the cause ofthe lesion in the cornea.
 8. A method of analyzing a corneal lesionusing an anterior segment image, the method comprising: a step ofacquiring, by an image acquisition unit, an anterior segment image froman eyeball of a subject; a step of extracting, by a feature extractor,feature information on a position and a cause of a lesion in a cornea ofthe eyeball from the anterior segment image by applying a convolutionlayer to the anterior segment image and a database in which clinicalinformation pre-acquired by analyzing positions and causes of lesions inthe corneas of subjects is stored; and a step of analyzing anddetermining, by a result determination unit, the position and the causeof the lesion in the cornea after identifying a position of the corneafrom the anterior segment image using the feature information.
 9. Themethod of claim 8, wherein the step of extracting, by the featureextractor, the feature information comprises: a step of extracting thefeature information including a multiple-channel feature map forextracting a suspicious region, from the anterior segment image, thefeature information being destined for a residual network (ResNet)obtained by stacking at least one network of a plurality of networkseach including a convolution layer, a pooling layer, and an activationfunction or a rectified linear unit (ReLU) function.
 10. The method ofclaim 9, wherein the step of extracting the feature informationcomprises: a step of extracting a lesion region more precise than thesuspicious region through convolution of the feature information andpositional data of a lesion in the clinical information, the positionaldata representing a position of the lesion, by a lesion attentionmodule.
 11. The method of claim 8, wherein the step of extracting, bythe feature extractor, the feature information comprises, a step ofadjusting, by a slit lamp mask adjustment module of the featureextractor, a slit beam image present in a region of the anterior segmentimage.
 12. The method of claim 11, wherein the step of adjusting, by theslit lamp mask adjustment module, the slit beam image comprises: a stepof applying, the slit lamp mask adjustment module, the convolution layerto the anterior segment image and an already-collected slit lamp regionmask; and a step of adjusting a masking ratio by applying a weightingfactor for a slit beam portion.
 13. The method of claim 8, wherein thestep of determining, by the result determination unit, the position andthe cause of the lesion in the cornea comprises: a step of outputting,by the feature extractor, a 3^(rd) label feature vector; a step ofoutputting a prediction vector corresponding to a 2^(nd) label featurevector computed by applying a fully connected layer to the 3^(rd) labelfeature vector; and a step of re-inputting, by the result determinationunit, the feature vector and the prediction vector to the fullyconnected layer.
 14. A computer-readable storage device on which aprogram that performs the method according to claim 8 is recorded.